Skip to content

Conversation

@QuantuMope
Copy link
Contributor

As of torch 2.6, torch.cuda.amp.autocast is now deprecated. This PR replaces all calls with torch.amp.autocast('cuda'). Confirmed that this is backward compatible with torch >= 2.1 which is used by CICD.

@QuantuMope QuantuMope requested a review from runjerry April 11, 2025 18:06
@runjerry
Copy link
Contributor

I feel a tricky thing is that some of our cluster docker images were built with older pytorch versions where torch has no amp attribute.

@QuantuMope
Copy link
Contributor Author

I feel a tricky thing is that some of our cluster docker images were built with older pytorch versions where torch has no amp attribute.

I think the oldest torch version is 2.1 on our cluster dockers, which I've confirmed works with this change.

Are there dockers that have older versions than this?

@runjerry
Copy link
Contributor

I feel a tricky thing is that some of our cluster docker images were built with older pytorch versions where torch has no amp attribute.

I think the oldest torch version is 2.1 on our cluster dockers, which I've confirmed works with this change.

Are there dockers that have older versions than this?

I think "horizonrobotics/cuda:11.3-cudnn8-py3.8-ubuntu20.04_kincpp" is using torch 1.11+cu11.3, this is the docker I am currently running for env v1, but I think I will switch to v2 soon. I also noticed some @Haichao-Zhang 's running jobs using this docker image, but not sure if he is still actively using it.

@QuantuMope
Copy link
Contributor Author

QuantuMope commented Apr 11, 2025

I feel a tricky thing is that some of our cluster docker images were built with older pytorch versions where torch has no amp attribute.

I think the oldest torch version is 2.1 on our cluster dockers, which I've confirmed works with this change.
Are there dockers that have older versions than this?

I think "horizonrobotics/cuda:11.3-cudnn8-py3.8-ubuntu20.04_kincpp" is using torch 1.11+cu11.3, this is the docker I am currently running for env v1, but I think I will switch to v2 soon. I also noticed some @Haichao-Zhang 's running jobs using this docker image, but not sure if he is still actively using it.

Got it. We can just hold off on this PR for now and merge it when we need it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants